Skip to content

Replace univocity parsers with FastCSV for CSV handling except for PSSE #3627

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Replace univocity parsers with FastCSV for CSV handling except for PSSE #3627

wants to merge 6 commits into from

Conversation

@rolnico
Copy link
Member

@rolnico rolnico commented Oct 23, 2025 •
edited
Loading

Please check if the PR fulfills these requirements

  • The commit message follows our guidelines
  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • A PR or issue has been opened in all impacted repositories (if any)

Does this PR already have an issue describing the problem?
No

What kind of change does this PR introduce?
Change of dependency

What is the current behavior?
PowSyBl uses univocity-parsers for handling CSV files (parsing and writing), but this lib is abandoned since 2021.

What is the new behavior (if this is a feature change)?
PowSyBl now uses FastCSV for handling CSV files.

Does this PR introduce a breaking change or deprecate an API?

  • Yes
  • No

If yes, please check if the following requirements are fulfilled

  • The Breaking Change or Deprecated label has been added
  • The migration steps are described in the following section

What changes might users need to make in their application due to this PR? (migration steps)

Other information:

PSS-E handling with be managed in a specific PR

rolnico added 6 commits October 9, 2025 14:56
@rolnico
Add FastCSV dependency to pom.xml for CSV processing
5d0af01 
Signed-off-by: Nicolas Rol <[email protected]>
@rolnico
Refactor StringAnonymizer to replace Univocity with FastCSV for CSV…
de76762 
… processing and improve resource management with try-with-resources.

Signed-off-by: Nicolas Rol <[email protected]>
@rolnico
Replace Univocity with FastCSV in IEEE CDF network parsing
c7bbbbc 
Signed-off-by: Nicolas Rol <[email protected]>
@rolnico
Refactor EuropeanLvTestFeederFactory to replace Univocity with FastCSV
87fbb12 
Signed-off-by: Nicolas Rol <[email protected]>
@rolnico
Refactor TimeSeries to replace Univocity with FastCSV for CSV proce…
3fd92e7 
…ssing.

Signed-off-by: Nicolas Rol <[email protected]>
@rolnico
Refactor AbstractCgmesAliasNamingStrategy to replace Univocity with…
1b1dacf 
… FastCSV for CSV writing

Signed-off-by: Nicolas Rol <[email protected]>
@rolnico rolnico self-assigned this Oct 23, 2025
@rolnico rolnico added dependencies Pull requests that update a dependency file Release 12/2025 labels Oct 23, 2025
@sonarqubecloud
Copy link

sonarqubecloud bot commented Oct 23, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
84.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@rolnico rolnico moved this from TODO to In Progress in Release 12/2025 Oct 30, 2025
@rolnico
Copy link
Member Author

rolnico commented Nov 3, 2025

I did some tests to compare the performances of this version vs main. Each tested operation was performed 50 times consecutively, and the results are based on the last 30 draws.

The operations tested where:

  • For the Anonymizer, I tested writeCsv and readCsv for 5000000 values anonymized.
  • For the Time Series, I tested the time series generation, TimeSeriesTable.writeCsv, and TimeSeries.parseCsv. The dataset consisted of 1000 double time series of 3000 time steps in 5 versions, which represents a CSV file of 290 Mo once exported.
  • For the IEEE CDF, I tested the IeeeCdfNetworkFactory.create33 and IeeeCdfNetworkFactory.create69 methods.

The results are as follows:

Operation main (ms) FastCSV (ms) Difference (ms) Difference (%)
Anonymizer - Write 311,393 181,985 -129,408 -41,56 %
Anonymizer - Read 2006,383 1971,176 -35,207 -1,75 %
Time Series - Generation 198,604 181,077 -17,527 -8,82 %
Time Series - Write 1818,053 1800,404 -17,649 -0,97 %
Time Series - Read 2614,167 2729,676 +115,509 +4,42 %
IEEE CDF - 33 7,210 0,849 -6,361 -88,22 %
IEEE CDF - 69 5,048 1,020 -4,028 -79,80 %

The results here are very encouraging: it seems that the change of dependency can optimize the performances in some cases.
However, since there is not much change in the Time series performances, it would be interesting so see if the time is spent in the library or in the code around it.

@rolnico rolnico marked this pull request as ready for review November 13, 2025 15:57
@rolnico rolnico moved this from In Progress to Waiting for review in Release 12/2025 Nov 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@rolnico rolnico

Labels

dependencies Pull requests that update a dependency file IEEE Release 12/2025 Time series

Projects

Release 12/2025
Status: Waiting for review

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rolnico